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Abstract 

We present results from the first geological field tests of the 'Cyborg Astrobiologist', which is a wearable com- 
puter and video camcorder system that we are using to test and train a computer- vision system towards having some 
of the autonomous decision-making capabilities of a field-geologist and field-astrobiologist. The Cyborg Astrobiol- 
ogist platform has thus far been used for testing and development of these algorithms and systems: robotic acqui- 
sition of quasi-mosaics of images, real-time image segmentation, and real-time determination of interesting points 
in the image mosaics. The hardware and software systems function reliably, and the computer-vision algorithms 
are adequate for the first field tests. In addition to the proof-of-concept aspect of these field tests, the main result 
of these field tests is the enumeration of those issues that we can improve in the future, including: first, detection 
and accounting for shadows caused by 3D jagged edges in the outcrop; second, reincorporation of more sophisti- 
cated texture-analysis algorithms into the system; third, creation of hardware and software capabilities to control the 
camera's zoom lens in an intelligent manner; and fourth, development of algorithms for interpretation of complex 
geological scenery. Nonetheless, despite these technical inadequacies, this Cyborg Astrobiologist system, consisting 
of a camera-equipped wearable-computer and its computer-vision algorithms, has demonstrated its ability of finding 
genuinely interesting points in real-time in the geological scenery, and then gathering more information about these 
interest points in an automated manner. 

Keywords: computer vision, robotics, image segmentation, uncommon map, interest map, field geology. Mars, 
wearable computers, co-occurrence histograms, gypsum, Miocene. 
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1 Introduction 

"With great autonomy comes great responsibility" 

-with apologies to Peter Parker's Uncle Ben (Lee et al. 2002, 2004). 

Outside of the Mars robotics community, it is commonly presumed that the robotic rovers on Mars are controlled 
in a time-delayed joystick manner, wherein commands are sent to the rovers several if not many times per day, as new 
information is acquired from the rovers' sensors. However, inside the Mars robotics community, they have learned 
that this process is rather cumbersome, and they have developed much more elegant methods for robotic control of 
the rovers on Mars, with highly significant degrees of robotic autonomy. Particularly, the Mars Exploration Rover 
(MER) team has demonstrated autonomy for the two robotic rovers Spirit & Opportunity to the level that: practically 
all commands for a given Martian day (1 'sol' = 24.6 hours) are delivered to each rover from Earth before the robot 
wakens from its power-conserving nighttime resting mode (Arvidson et al. 2003; Maki et al. 2003; Bell et al. 2003; 
Squyres et al. 2003; Crisp et al. 2003). Each rover then follows the commanded sequence of moves for the entire sol, 
moving to desired locations, articulating its arm with its sensors to desired points in the workspace of the robot, and 
acquiring data from the cameras and chemical sensors. From an outsider's point of view, these capabilities may not 
seem to be significantly autonomous, in that all the commands are being sent from Earth, and the MER rovers are 
merely executing those commands. But upon closer inspection, what the MER team has achieved is truly amazing: 

• Firstly, the rovers can move to points 50 — 150 meters away in one sol with autonomous obstacle avoidance 
enabled for the uncertain or dangerous parts of the journey (Goldberg, Maimone & Matthies 2002; Nesnas, Maimone 
& Das 1999). 

• Secondly, prior to a given sol, based upon information received after the previous sol, the MER team has the 
remarkable capabilities to develop a command sequence of tens or hundreds of robotic commands for the entire sol. 
As of July 4, 2004, this was taking 4-5 hours per sol for the mission team to complete, rather than the 17 hours per sol 
that it took at the beginning of the MER missions (Squyres, unpublished', 2004). 

Such capabilities for semi-autonomous teleoperated robotic 'movement and discovery' are a significant leap be- 
yond the capabilities of the previous Mars lander missions of Viking I & II and of Pathfinder & Sojourner, particularly 
in the large improvements of the mobility of both the cameras and the instrumentation onboard the robot, allowing 
significant discoveries to be made, and new insights to be obtained (Chan et al. 2004; Squyres et al. 2004; Squyres 
& Athena Science Team 2004; Moore 2004; Cathng 2004; Crumpler et al. 2004; Golembek et al. 2004; Squyres 
2004; McSween et al. 2004; Christensen et al. 2004; Arvidson et al. 2004; Greeley et al. 2004; Herkenhoff et al. 
2004; Bell et al. 2004). Nonetheless, we would like to build upon this great success of the MER rovers by developing 
enhancing technology that could be deployed in future robotic and/or human exploration missions to the Moon, Mars, 
and Europa. 

One future mission deserves special discussion for the technology developments described in this paper: the Mars 
Science Laboratory, planned for launch in 2009 (MSL'2009). A particular capability desired for this MSL'2009 
mission will be to rapidly traverse to up to three geologically-different scientific points-of-interest within the landing 
ellipse (Heninger et al.^, unpublished, 2003). These three geologically-different sites will be chosen from Earth by 
analysis of relevant remote-sensing imagery from a possible co-launched, satellite-based. Synthetic Aperture Radar 
(SAR) system or high-resolution maps from prior missions (i.e. 30 cm/pixel spatial resolution from the HiRISE imager 
on the Mars Reconnaisance Explorer with imagery expected in 2006-2007, McEwen et al. 2003). Possible desired 

'lecture notes from the course on the MER rovers from Astrobiology Summer School "Planet Mars", Universidad Intemacional Menendez 
Pelayo Santander, Spain 

^Heninger, B., M. Sander, J. Simmonds, F. Palluconi, B. Muirhead, C. Whetsel, "Mars Science Laboratory Mission 2009 Landed Science 
Payload", Proposal Information Package (draft version), |http://centauri.larc.nasa.gov/msl/PlP-Drft_FBO-RevA-03lll3.pdf| 
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maximal traversal rates could range from 300-2000 meters/sol in order to reach each of the three points-of-interest 
in the landing ellipse in minimum time. These are our rough estimates of the typical maximum traversal rates, based 
upon current capabilities, mission goals, increased rover size and expectations of the state of technology for the 
possible 2009 mission; our estimate is partially based upon such work as described in: Goldberg, Maimone & Matthies 
(2002); Olson, Matthies, Schoppers & Maimone (2003); Crawford (unpublished^, 2002); and Crawford & Tamppari 
(unpublished^, 2002). 

Given these substantial expected traversal rates of the MSL'2009 rover, autonomous obstacle avoidance (Goldberg, 
Maimone & Matthies 2002) and autonomous visual odometry & localization (Olson, Matthies, Schoppers & Maimone 
2003) will be essential to achieve these rates, since otherwise, rover damage and slow science-target approach would be 
the results. Given such autonomy in the rapid traverses, it behooves us to enable the autonomous rover with sufficient 
scientific responsibility. Otherwise, the robotic rover exploration system might drive right past an important scientific 
target-of-opportunity along the way to the human-chosen scientific point-of-interest. Crawford & Tamppari (previous 
footnote, 2002) and their team summarize possible 'autonomous traverse science', in which every 20-30 meters during 
a 300 meter traverse (in their example), science pancam and Mini-TES (Thermal Emission Spectrometer) image 
mosaics are autonomously obtained. They state that "there may be onboard analysis of the science data from the 
pancam and the mini-TES, which compares this data to predefined signatures of carbonates or other targets of interest. 
If detected, traverse may be halted and information relayed back to Earth." This onboard analysis of the science 
data is precisely the technology issue that we have been working towards solving. This paper is the first detailed 
published report describing our progress towards giving a robot some aspects of autonomous recognition of scientific 
targets-of-opportunity (see McGuire et al. (2004) for a brief description and motivation of our work). 




Figure 1 : On the left, Diaz Martinez & McGuire with the Cyborg Astrobiologist System at 1 :30PM on 3 March 2004, 
10 meters from the outcrop cliff that is being studied during the first geological field mission near Rivas Vaciamadrid. 
We are taking notes prior to acquiring one of our last-of-the-day mosaics and its set of interest-point chips. On the 
right is a close-up view of the same scene, but focusing on the half-seated/half-tripod Cyborg Astrobiologist and the 
assisting human Geologist. Photo copyright: Diaz Martinez, Ormo & McGuire 

Our general aim of study for this project is to develop autonomous computer vision algorithms for natural terrestrial 
geological scenery in uncontrolled lighting conditions (see our 'Cyborg Astrobiologist' research platform in Figure 
1), with the expectation that such algorithms will mature in the future so that they can be used in the human or robotic 
astrobiological exploration of Mars and some of the moons of our solar system. This project is not only armed at giving 
more scientific autonomy to robotic explorers of the surfaces of other planetary bodies, but it is also meant to take some 
of the burden off of the large team of human geologists at mission control here on the Earth by performing some of the 

^"NASA Applications of Autonomy Technology", NASA Ames Research Center, InteUigent Data Understanding Seminar, 
|http://is. arc. nasa.gov/IDU/slides/Crawford_Aut02c.pdf 

^"Mars Science Laboratory - Autonomy Requirements Analysis", NASA Ames Research Center, IntelUgent Data Understanding Seminar, 
|5tp://is.arc.nasa.gov/IDU/slides/Crawford_MSL02c.pd^ 
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low-level geological image analysis autonomously, thus freeing the human geologists to do more high-level analysis 
of the scenery. Such a system could also assist human geologists here in the field on the Earth for their studies, as well 
as astronauts in their geological field studies (who may be less-practiced in geology and more burdened with other 
duties and concerns). Our long-term aim of study is for the Cyborg Astrobiologist system to be able to understand the 
geological history of an outcrop in order to decide which area should be approached for sampling. Our short-term aim 
of study is to demonstrate that the Cyborg Astrobiologist system can (in the field) find the most unique regions in an 
image mosaic, and then to autonomously study those interesting regions in more detail. 

In this report, we give additional introductory background in Section 1 . 1 , a discussion of the hardware and software 
for our "Cyborg Astrobiologist" system in Section 2, summaries & results of the first two geological expeditions of 
our Cyborg Astrobiologist system in Sections 3 & 4, followed by a more general discussion & conclusions in Section 
5. 

1.1 Autonomous Recognition of Scientific Astrobiological Targets-of -Opportunity 

We discuss here two of the related efforts in the development of autonomous recognition of scientific targets-of- 
opportunity for astrobiological exploration: firstly, the work on developing a Nomad robot to search for meteorites in 
Antartica led by the Carnegie Mellon University Robotics Institute (CMU-RI) (Apostolopoulos et al. 2000; Pederson 
2001; Wagner 2000; Vandapel et al. 2000), and secondly, the work by a group at NASA Ames Research Center 
(NASA-ARC) on developing a Geological Field Assistant (GFA) (Guhck et al. (2001, 2002a, 2002b, 2003, 2004)). 
There are several other efforts in the community which also deserve mention (including Volpe 2003; Huntsberger et 
al. 2002; Storrie-Lombardi, Grigolini et al. 2002; Corsetti & Storrie-Lombardi 2003; Cabrol et al. 2001; Wettergreen 
et al. 1999; Whittaker et al. 1997; Cheeseman et al. 1988; Cheeseman & Stutz 1996). Outside of the astrobiology 
community, there is some similar work on automated obstacle avoidance (instead of target discovery) by Batavia & 
Singh (2001), among others. 

The CMU-RI team has programmed the capabilities to autonomously distinguish between rocks and ice/snow 
in various lighting conditions (direct sunlight, shadows, and diffuse light in overcast conditions) into the science 
computer of its meteorite-searching Nomad robot (Apostolopoulos et al. 2000). This autonomous decision between 
whether a pixel is ice/snow or if it is rock is decided by how 'blue' the pixel is: if the pixel has its 'blue' ratio 
(blue/ {blue + green)) above a certain lighting-dependent threshold, then it is considered to be ice/snow; otherwise, 
it is considered to be rock/meteorite. This system works rather well except when the rocks are in shadows (but we 
naively suggest that a non-linear separator in the 2D space of 'blue' and intensity could solve this problem even in 
shadows). After a rock/meteorite is segmented from the ice/snow based upon how 'blue' it is, it can then trigger 
the Nomad robot's Bayesian classifier network system to compute the probability of the rock being any one of six 
different meteor types or of 19 different terrestrial rock types, based upon data from a metal detector, from a color 
camera detector, and from a spectrometer. The Nomad robot "[autonomously] found and [autonomously] classified 
five indigenous meteorites [of 42 rocky targets studied and classified over a 2500 m^ search area] during an expedition 
to a remote site of Elephant Moraine [in Antarctica] in January 2000." (Apostolopoulos et al. 2000). 

The NASA- ARC GFA team have developed a number of algorithms for assisting an astronaut in doing geology or 
in adding capabilities to an autonomous robotic geologist (Gulick et al. (2001, 2002a, 2002b, 2003, 2004)). They have 
been building an image and spectral database of rocks and minerals [now there are over 700 such rock and mineral 
entries in their database] for continued development of advanced classifier algorithms, including future development 
work to autonomously identify 'key igneous, sedimentary and some metamorphic rocks' (Gulick et al. 2003). As of 
2003, they were able to identify "some igneous rocks from texture and color information; quartz, silica polymorphs, 
calcite, pyroxene and jarosite from both visible/near-infrared and mid-infrared spectra; and high-iron pyroxenes and 
iron-bearing minerals using visible/near-infrared spectra only" (Gulick et al. 2003). As of 2004, the NASA Jet Propul- 
sion Laboratory (JPL) CLARAty system team (Volpe 2004) had incorporated the NASA- ARC GFA team's rock, layer 
and horizon detectors (Gulick et al. 2001) for "possible use by MSL'2009 and [other] future Mars robotics missions" 
(Gulick et al. 2004). Work reported in 2004 includes significant progress on their igneous rock detection system: the 
Decision Tree method has yielded better and more decipherable results than the Bayesian Network approach, giving 
"correct identification of greater than 80% of granites and granodiorites and greater than 80% of andesites and basalts. 
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using color and texture algorithms combined [without spectral information]" (Gulick et al. 2004). 

2 The Cyborg Geologist & Astrobiologist System 

Our ongoing effort in this area of autonomous recognition of scientific targets-of-opportunity for field geology and 
field astrobiology is beginning to mature as well. To date, we have developed and field-tested a GFA-like "Cyborg 
Astrobiologist" system (McGuire et al. 2004) that now can: 

• Use human mobiUty to maneuver to and within a geological site and to follow suggestions from the computer as to 
how to approach a geological outcrop; 

• Use a portable robotic camera system to obtain a mosaic of color images; 

• Use a 'wearable' computer to search in real-time for the most uncommon regions of these mosaic images; 

• Use the robotic camera system to re-point at several of the most uncommon areas of the mosaic images, in order to 
obtain much more detailed information about these 'interesting' uncommon areas; 

• Use human intelligence to choose between the wearable computer's different options for interesting areas in the 
panorama for closer approach; and 

• Repeat the process as often as desired, sometimes retracing a step of geological approach. 

In the Mars Exploration Workshop in Madrid in November 2003, we demonstrated some of the early capabilities 
of our 'Cyborg' Geologist/Astrobiologist System (McGuire et al. 2004). We have been using this Cyborg system as 
a platform to develop computer- vision algorithms for recognizing interesting geological and astrobiological features, 
and for testing these algorithms in the field here on the Earth. 

The half-human/half-machine 'Cyborg' approach (see Figures 1 & 5) uses human locomotion and human-geologist 
intuition/intelligence for taking the computer vision-algorithms to the field for teaching and testing, using a wearable 
computer. This is advantageous because we can therefore concentrate on developing the 'scientific' aspects for au- 
tonomous discovery of features in computer imagery, as opposed to the more 'engineering' aspects of using computer 
vision to guide the locomotion of a robot through treacherous terrain. This means the development of the scientific 
vision system for the robot is effectively decoupled from the development of the locomotion system for the robot. 

After the maturation of the computer- vision algorithms, we hope to transplant these algorithms from the Cyborg 
computer to the on-board computer of a semi-autonomous robot that wiU be bound for Mars or one of the interesting 
moons in our solar system. 

2.1 Geological Approach to an Outcrop; Metonymical Contacts 

This method of using a semi-autonomous system to guide a geological approach to an outcrop, in which the wearable 
computer decides or gives options as to what in the panorama is the most interesting and deserves closer study, is 
meant to partially emulate the decision process in the mind of a professional field geologist. An approach from a 
distance, using information available at further distances to guide the approach to closer distances, is a reasonable, and 
possibly, generic method used by most field geologists when approaching an outcrop subject to study. Our technique 
of presenting several point-like options for more close-up study is a reasonable method to 'Unearize' the convoluted 
thought processes of a practicing field geologist. 

McGreevy (1992, 1994) has completed significant detailed cognitive & ethnographic studies of the practice of 
field geology, using a head-mounted camera and virtual-reality visor system for part of his study as well as extensive 
on-site interviews of field-geologist colleagues. One particularly interesting concept from McGreevy's work is that 
of 'metonymic' geological contacts. For field-geology, this concept of metonymy is the juxtaposition of differing 
geological units as well as the relation between them; metonymy is to differences, as metaphor is to similarities. 

Both of the field geologists on our team have also independently stressed the importance to field geologists of such 
geological contacts and the differences between the geological units that are separated by the geological contact. For 
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Figure 2: An image segmentation made by geologist, Diaz Martinez, of the outcrop during the first mission to Rivas 
Vaciamadrid. Region 1 has a tan color and a blocky texture; Region 2 is subdivided by a vertical fault and has more 
red color and a more layered texture than Region 1 ; Region 3 is dominated by white and tan layering; and Region 
4 is covered by vegetation. The dark & wet spots in Region 3 were only observed during the second mission, 3 
months later. The Cyborg Geologist/Astrobiologist made its own image segmentations for portions of the cliff face 
that included the area of the white layering at the bottom of the cliff (see Figures 6-8). Photo copyright: Diaz Martinez, 
Ormo & McGuire 

this reason, in March 2003, the roboticist and the field geologists decided that the most important tool to develop for the 
beginning of our computer vision algorithm development was that of 'image segmentation' . Such image segmentation 
algorithms would allow the computer to break down a panoramic image into different regions (see Figure 2 for an 
example), based upon similarity, and to find the boundaries or contacts between the different regions in the image, 
based upon difference. Much of the remainder of this paper discusses the first geological field trials with the wearable 
computer of the segmentation algorithm we have developed over the last year. 

2.2 Image Segmentation, Uncommon Maps, Interest Maps, and Interest Points 

With human vision, a geologist; 

• Firstly, tends to pay attention to those areas of a scene which are most unlike the other areas of the scene; and then, 

• Secondly, attempts to find the relation between the different areas of the scene, in order to understand the 
geological history of the outcrop. 
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The first step in this prototypical thought process of a geologist was our motivation for inventing the concept 
of uncommon maps. See Figure 3 for a simple illustration of the concept of an uncommon map. We have not 
yet attempted to solve the second step in this prototypical thought process of a geologist, but it is evident from the 
formulation of the second step, that human geologists do not immediately ignore the common areas of the scene. 
Instead, human geologists catalog the common areas and put them in the back of their minds for "higher-level analysis 
of the scene", or in other words, for determining explanations for the relations of the uncommon areas of the scene 
with the common areas of the scene. 



■ 

I 



Figure 3: For the simple, idealized image on the left, we show the corresponding uncommon map on the right. The 
whiter areas in the uncommon map are more uncommon than the darker areas in this map. 

Prior to implementing the 'uncommon map', the first step of the prototypical geologist's thought process, we 
needed a segmentation algorithm, in order to produce pixel-class maps to serve as input to the uncommon map algo- 
rithm. We have implemented the classic co-occurrence histogram algorithm (Haralick, Shanmugan & Dinstein 1973; 
Haddon & Boyce 1990). For this work, we have not included texture information in the segmentation algorithm and 
the uncommon map algorithm. Currently, each of the three bands of HSI color information is segmented separately 
(see Section 2.4 and the appendix), and later merged in the interest map by summing three independent uncommon 
maps. Maybe in future work, image segmentation simultaneously using color & texture could be developed for and 
tested on the Cyborg Astrobiologist System (e.g., Freixenet, Munoz, Marti & Llado 2004). 

The concept of an 'uncommon map' is our invention, though it probably has been independently invented by 
other authors, since it is somewhat useful. In our implementation, the uncommon map algorithm takes the top 8 
pixel classes determined by the image segmentation algorithm, and ranks each pixel class according to how many 
pixels there are in each class. The pixels in the pixel class with the greatest number of pixel members are numerically 
labelled as 'common', and the pixels in the pixel class with the least number of pixel members are numerically labelled 
as 'uncommon'. The 'uncommonness' hence ranges from 1 for a common pixel to 8 for an uncommon pixel, and we 
can therefore construct an uncommon map given any image segmentation map. Rare pixels that belong to a pixel class 
of 9 or greater are usually noise pixels in our tests thus far, and are currently ignored. In our work, we construct several 
uncommon maps from the color image mosaic, and then we sum these uncommon maps together, in order to arrive at 
a final interest map. This summing to determine an interest map has been also studied by Rae, Fislage & Ritter (1999). 
In our implementation for this paper, the interest map is a convenient name to call the sum of several uncommon maps, 
so the interest map is in a sense also an uncommon map. However, in general, the interest map could also sum other 
maps as well; and in the implementation of Rae, Fislage & Ritter (1999), their 'cortical' interest map algorithm not 
only sums several different types of maps, but the algorithm also adapts the weighting coefficients for these maps in 
order to highlight the summand map(s) which are most salient, most uncommon, or most interesting. For our work, 
we have not activated this adaptive summing of the interest map algorithm, and we only use uncommon maps in the 
interest map sum. 
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In this paper, we develop and test a simple, high-level concept of interest points of an image, which is based upon 
finding the centroids of the smallest (most uncommon) regions of the image. Such a 'global' high-level concept of 
interest points differs from the lower-level 'local' concept of interest points (Forstner 1986; Forstner & Giilch 1987) 
based upon corners and centers of circular features. However, this latter technique with local interest points is used 
by the MER team for their stereo-vision image matching and for their visual-odometry and visual-localization image 
matching (Arvidson et al. 2003; Goldberg, Maimone & Matthies 2002; Olson, Matthies, Schoppers & Maimone 2003; 
Nesnas, Maimone & Das 1999). Our interest point method bears somewhat more relation to the higher-level wavelet- 
based salient points technique (Sebe, Tian, Loupias, Lew & Huang 2003), in that they search first at coarse resolution 
for the image regions with the largest gradient, and then they use wavelets in order to zoom in towards the salient 
point within that region that has the highest gradient. Their salient point technique is edge-based, whereas our interest 
point is currently region-based. Since in the long-term, we have an interest in geological contacts, this edge-based 
& wavelet-based salient point technique could be a reasonable future interest-point algorithm to incorporate into our 
Cyborg Astrobiologist system for testing. 

For a review of some of the general aspects of computer vision for natural scenery see: Batlle, Casals, Freixenet & 
Marti (2000). We give a description of our specific software system in Section 2.4 and more details in the Appendix. 




Figure 4: Map of the three tripod positions of the Cyborg Astrobiologist system during the second mission to Rivas 
Vaciamadrid (this map is not to scale). See Section 3.3 for a description of the geological approach suggested by 
the Cyborg Astrobiologist to the outcrop. See Figure 1 for an image of the system in action during the first mission, 
when it was near the 2nd tripod position shown here. See Figure 5 for an image of the system in action during the 
first mission when it was near the 1st tripod position shown here. Figures 6, 7 & 8 show mosaics obtained during the 
second mission for the 3rd, 1st & 2nd (respectively) tripod positions shown here; these same Figures (6, 7 & 8) also 
show the higher resolution color imagery of the top three interest points for each of the image mosaics. 
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2.3 Hardware for the Cyborg Astrobiologist 

The non-human hardware of the Cyborg Astrobiologist system consists of: 

• a 667 MHz wearable computer (from ViA Computer Systems in Minnesota) with a 'power-saving' Transmeta 
'Crusoe' CPU and 112 MB of physical memory, 

• an SV-6 Head Mounted Display (from Tekgear in Virginia, via the Spanish suppUer Decom in Valencia) with native 
pixel dimensions of 640 by 480 that works well in bright sunlight, 

• a SONY 'Handycam' color video camera (model DCR-TRV620E-PAL), 

• a thumb-operated USB finger trackball from 3G Green Green Globe Co., resupphed by ViA Computer Systems and 
by Decom, 

• a small keyboard attached to the human's arm, 

• a tripod for the camera, and 

• a Pan-Tilt Unit (model PTU-46-70W) from Directed Perception in California with a bag of associated power and 
signal converters. 

The power-saving aspect of the wearable computer's Crusoe processor is important because it extends battery 
hfe, meaning that the human does not need to carry very many spare batteries, meaning less fatigue for the human 
geologist. A single Uthium-ion battery, which weighs about 1kg, can last for 3 hours or so for this application. The 
SONY Handycam provides real-time imagery to the wearable computer via an 1EEE1394/Firewire communication 
cable. The current system uses five independent batteries, one for each sub-system: the computer, the head-mounted 
display, the camera, the Pan-Tilt Unit, and the communication interface between the Pan-Tilt Unit and the computer. 
Assuming fully-charged batteries, all batteries are currently more than sufficient for a mission of several hours. 

The main reason for using the tripod is that it allows us to use the Pan-Tilt Unit for mosaicking and for re- 
pointing. Without the tripod, there would be an unusual amount of jitter in the image-registration caused by the human 
fidgeting with the camera, regardless of whether the camera is hand-mounted, shoulder-mounted or head-mounted. 
In future studies, if the mosaicking or re-pointing is not desired, then neither the Pan-Tilt Unit nor the tripod would 
be necessary; eUminating both the tripod and the Pan-Tilt Unit would certainly add to the mobility of the Cyborg 
Astrobiologist system. 

For this particular study, which included the Pan-Tilt Unit and the tripod, the wearable computer could have been 
replaced by a suitable non-wearable computer. However, we would Uke to retain flexibility in our work for future 
studies, in which the mobility granted by the wearable computer may be essential. The head-mounted display was 
much brighter than our earlier tablet display, and together with the thumb-operated finger mouse, the head-mounted 
display was more ergonomic than than our earlier tablet display and stylus. However, the spatial resolution of the 
head-mounted display was somewhat less than the resolution of the tablet display, but sufficient for the current task. It 
would be useful to be able to switch rapidly between the tablet display and the head-mounted display, but our system 
is not capable of this at this moment. 

The thumb-operated finger trackball worked rather well for navigating the Microsoft Windows environment of the 
wearable system. However, when used outdoors in bright sunlight, we often had to shade the trackball with the user's 
second hand in order for it to function properly. Since this mouse is an optical mouse, the mouse fails when sufficient 
sunlight leaks into the trackball. Alternatively, we could have sealed the seams in the plastic mouse with tape or glue. 
Additionally, since the wearable computer only has one USB port, the system would have been easier to use when 
downloading data to a USB memory stick if the USB finger trackball had been the PS2 model of the finger trackball; 
this will be changed before the next mission. 

Two other issues affected the usage & mobility of the wearable system in the field. First, the serial port of the 
ViA wearable computer often failed to provide a signal of sufficient amplitude; this signal is needed to drive the 
Pan-Tilt Unit used for robotic exploration of the environment. We sent the wearable computer system back to the 
manufacturer for investigation, and they suggested that we reinstall the COM port in software when we had this 
problem. But since this problem was so frequent (especially after putting the wearable computer into sleep mode for 
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power conservation), we decided to fix the signal problem with an external Operational Amplifier (OpAmp). This 
external OpAmp system functions well, but it requires occasional 9V battery changes, and it needs a certain amount 
of engineering improvements in ergonomics and miniaturization. Second, the Pan-Tilt Unit requires +12 to +30V of 
power The battery we have chosen for this task weighs about 2kg, and lasts for 5-10 hours with the currently-used 
Cyborg software. The battery, together with the signal conversion box (from Serial signals to PTU signals) and the 
signal OpAmp box, require an extra, small carrying bag in order to use the PTU in the field. This bag is usually placed 
on the ground when each tripod position is reached, but for transport between the tripod positions the PTU bag is 
added baggage and puts some minor limits on mobility. The PTU system certainly merits further engineering. 

2.4 Software for the Cyborg Astrobiologist 

The wearable computer processes the images acquired by the color digital video camera, to compute a map of interest- 
ing areas. The computations include: simple mosaicking by image-butting, as well as two-dimensional histogramming 
for image segmentation (Haralick, Shanmugan & Dinstein 1973; Haddon & Boyce 1990). This image segmentation is 
independently computed for each of the Hue, Saturation, and Intensity (H,S,1) image planes, resulting in three different 
image-segmentation maps. These image-segmentation maps were used to compute 'uncommon' maps (one for each 
of the three (H,S,1) image-segmentation maps): each of the three resulting uncommon maps gives highest weight to 
those regions of smallest area for the respective (H,S,1) image planes. Finally, the three (H,S,1) uncommon maps are 
added together into an interest map, which is used by the Cyborg system for subsequent interest-guided pointing of 
the camera. The image-processing algorithms and robotic systems-control algorithms are all programmed using the 
graphical programming language, NEO/NST (Ritter et al.^, unpublished, 1992, 2002). Using such a graphical pro- 
gramming language adds certain flexibility and ease-of-understanding to our Cyborg Astrobiologist project, which is 
by its nature largely a software project. We discuss some of the details of the software implementation in the appendix. 

After segmenting the mosaic image (see the appendix and Figure 9), it becomes obvious that a very simple method 
to find interesting regions in an image is to look for those regions in the image that have a significant number of 
uncommon pixels. We accomplish this by (Figure 10 for an example): first, creating an uncommon map based upon a 
linear reversal of the segment peak ranking (we have also studied nonlinear reversals of the segment ranking); second, 
adding the 3 uncommon maps (for H, S, & /) together to form an interest map; and third, blurring this interest map*. 

Our study extends Rae, Fislage & Ritter's interest map concept by using uncommon maps based upon image seg- 
mentation of H, S & I for the component maps of the interest map, rather than using the H, S & I maps themselves 
for the component maps of the interest map. This extension is especially important for our purposes of finding the most 
uncommon and interesting areas of an image. By using such an image-fusion technique as Rae, Fislage & Ritter's Cor- 
tical Interest Map technique to combine three uncommon/segmentation maps we also avoided the complexity of devel- 
oping a true image-segmentation algorithm for color images in order to determine a single uncommon/segmentation 
map. If the current work is sufficiently valued by the Astrobiology or other communities, then maybe the Cyborg 
platform could be used for the development, testing, and optimization of true color image-segmentation algorithms. 

Based upon the three largest peaks in the blurred/smoothed interest map, the Cyborg system then guides the Pan- 
Tilt Unit to point the camera at each of these three positions to acquire high-resolution color images of the three 
interest points, as shown in four different examples in Figures 6-8. The robotic control and image mosaicking portion 
of the Cyborg software required significant time for development and debugging since the logic of this autonomy is 
somewhat complicated. However, by extending a simple image-acquisition and image-processing system to include 
robotic and mosaicking elements, we were able to conclusively demonstrate that the system can make reasonable 
decisions by itself in the field for robotic pointing of the camera (Figures 6-8). 

^Ritter, H., et al. (1992, 2002). "The Graphical Simulation Tookit, Neo/NST", |!ittp://www.TecbFak.Uni-Bielef6ld.DE/teclitak/ags/iii71 
^with a gaussian smoothing kernel of width B = 10. This smoothing kernel effectively gives more weight to clusters of uncommon pixels, 
rather than to isolated, rare pixels. 
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3 Descriptive Summaries of the Field Site and of the Expeditions 

On the 3rd of March, 2004, three of the authors (McGuire, Diaz Martinez & Ormo) tested the "Cyborg Astrobiol- 
ogist" system for the first time at a geological site, the gypsum-bearing southward-facing stratified cUffs near the 
"El Campillo" lake of Madrid's Southeast Regional Park (Castilla Canamero', unpublished, 2001; Calvo, Alonso & 
Garcia del Cura 1989; IGME 1975), outside the suburb of Rivas Vaciamadrid. 

After post-mission analysis of the results from the Cyborg Astrobiologist's first mission, a second mission was 
planned for the 12th of May, 2004. After problems were discovered in preparation for the second mission, we decided 
to delay the second mission to Rivas Vaciamadrid until the 1 1th of June, 2004. For the second, rescheduled mission 
on the 1 1th of June, McGuire & Ormo took the Cyborg Astrobiologist system to the same southward-facing stratified 
cliffs outside of Rivas. Due to the significant storms in the previous 3 months, there were 2 points in the gypsum cliffs 
that were leaking water down the face of the cliff (despite the hot weather before & during the second mission). 

3.1 Description and Choice of the Field Site at Rivas Vaciamadrid 

The rocks at the outcrop are of Miocene age (15-23.5 Myrs before present), and consist of gypsum and clay minerals, 
also with other minor minerals present (anhydrite, calcite, dolomite, etc.). These rocks form the whole cliff face 
which we studied, and they were all deposited in and around lakes with high evaporation (evaporitic lakes) during the 
Miocene. They belong to the so-called Lower Unit of the Miocene. Above the cliff face is a younger unit (which we 
did not study) with sepiolite, chert, marls, etc., forming the white relief seen above from the distance, which is part 
of the so-called Middle Unit or Intermediate Unit of the Miocene. The color of the gypsum is normally grayish, and 
in-hand specimens of the gypsum contain large and clearly visible crystals. 
We chose this locality for several reasons, including: 

• It is a so-called 'outcrop', meaning that it is exposed bedrock (without vegetation or soil cover); 

• It has distinct layering and significant textures, which will be useful for this study and for future studies; 

• It has some degree of color differences across the outcrop; 

• It has tectonic offsets, which will be useful for future high-level analyses of searching for textures that are 
discontinuously offset from other parts of the same texture; 

• It is relatively close to our workplace; 

• It is not a part of a noisy and possibly dangerous construction site or highway site. 

The upper areas of the chosen portion of the outcrop at Rivas Vaciamadrid (see Figure 2) were mostly of a tan 
color and a blocky texture. There was some faulting in the more layered middle areas of the outcrop, accompanied 
by some slight red coloring. The lower areas of the outcrop were dominated by white and tan layering. Due to some 
differential erosion between the layers of the rocks, there was some shadowing caused by the reUef between the layers. 
However, this shadowing was perhaps at its minimum for direct lighting conditions, since both expeditions were taken 
at mid-day and since the outcrop runs from East to West. Therefore, by performing our field-study at Rivas during 
mid-day, we avoided possible longer shadows from the sun that might occur at dawn or dusk. 

Diaz Martinez had studied similar geological sites in the region of Rivas Vaciamadrid, prior to the missions. He 
explains the bright white area at the bottom of the cliffs as being due to mineral efflorescences (most probably calcium 
and magnesium sulfates) on the surface of the cliff rock. This cliff rock is mostly gypsum and clays. The efflorescence 
is just a surface feature formed by evaporation of the water seeping out from the rock pores by capillarity. 

'Castilla Canamero, G. (2001). Infome sobre las practicas profesionales realizadas en el Centro de Educacidn Ambiental: El Campillo. Internal 
report to the Consejeria de Medio Ambiente de la Comunidad de Madrid. 
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3.2 Details from the First Mission to Rivas 

We arrived at the site at 10:30am on March 3, and for the next hour, the two geologists of our team both made inde- 
pendent expert-geologist assessments of the area. In their assessments, these two geologists included their reasoning 
processes of how they would gradually approach and study the site, from an initial distance of about 300 meters down 
to a final distance of physical contact with the geological structures. An analysis of the geologists' assessments will 
be made available in the near future. However, in Figure 2, we show the segmentation of the outcrop, according to the 
geologist Diaz Martinez, for reference and as a sample of the geologists' assessments. 

At about 1 1:30am on the same day, the roboticist finished setting up the Cyborg Astrobiologist system, and started 
taking mosaic images of the cUff face, with the computer-controlled & motorized color video camera. It generally 
takes some time to set up the system (in this case one hour) because: 

• Several cables need to be attached and secured, 

• The computer needs to boot, 

• Three programs need to be activated and checked, 

• Communication between the serial port and the Pan-Tilt Unit needs to be confirmed, and finally 

• The non-computer-controlled zoom factor on the camera-lens needs to be set and optimized for the geological 
scenery, and the Pan-Tilt Unit's step size needs to be set to match the camera's zoom factor. 

The computer was worn on the user's belt, and typically took 3-5 minutes to acquire and compose a mosaic 
image composed of M x N subimages. Typical values of M x N used were 3x9 and 11x4. The sub-images 
were downsampled in both directions by a factor of 4-8 during these tests; the original sub-image dimensions were 
360 X 288. 

Several mosaics were acquired of the cUff face from a distance of about 300 meters, and the computer auto- 
matically determined the three most interesting points in each mosaic. Then, the wearable computer automatically 
repointed the camera towards each of the three interest points, in order to acquire non-downsampled color images 
of the region around each interest point in the image. All the original mosaics, all the derived mosaics and all the 
interest-point subimages were then saved to hard disk for post-mission study. During the entire expedition, the robotics 
engineer could observe what the wearable computer system was doing through a very bright, sunhght-readable head- 
mounted computer display (HMD). The robotics engineer could & did make adjustments on-the-fly to the automated- 
exploration settings and algorithms by using this HMD in conjunction with a hand-held free-movement thumbmouse 
and a small arm-mounted keyboard. 

Three other tripod positions were chosen for acquiring mosaics and interest-point image-chip sets. At each of the 
three tripod positions, 2-3 mosaic images and interest-point image-chip sets were acquired. One of the chosen tripod 
locations was about 60 meters (see Figure 5) from the cliff's face; the other two were both about 10 meters (see Figure 
1) from the cliff face. There was some concern that if we approached much closer for any significant amount of time, 
then rocks from the fragile gypsum-bearing cliffs could fall upon us (there was plenty of evidence of past rock falls). 
We kept a distance, but noticed no such rock falls during the time we were there. 

3.3 Details from the Second Mission to Rivas 

Ormo & McGuire arrived at 12pm on June 11. One of us first suspected that somebody had Ut some fires near the 
bottom of the cliffs, since there were two black stains, which were medium-sized and soot-like, covering in total about 
10-15% (about 4-6 m^) of the bright white area at the bottom of the cliffs. Closer analysis discounted this suspicion, 
as the black stains were really caused by two separate leaks of water^ from the cUff face, with each leak wetting & 
darkening a small region of the cliff face. 

Due to the hot sun, we chose our first Cyborg Astrobiologist tripod position (see the map in Figure 4) to be in the 
cool shade of the grove of trees about 60 meters from the cUff face (See Figure 5). We stayed at this tripod position 

* audible from 60 meters distance. 
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for about 1.3 hours, acquiring several 2x2 mosaics, and several 4x3 mosaics, with image-downsamphng of 4. The 
system most often determined the wet spots (Figure 7) to be the most interesting regions on the cliff face. This was 
encouraging to us, because we also found these wet spots to be the most interesting regions^. 

To end the mission, we chose two other tripod positions (see the map in Figure 4) for the Cyborg Astrobiologist to 
acquire mosaics, to search for interest points and then to record detailed information about the interest points. The first 
of these two tripod positions was about 10 meters from the cliff face, which was significantly closer than the original 
tripod position in the grove of trees and was approximately the same position as in the first mission, as depicted in 
Figure 1. During this 'close-up' study of the cUff face, we intended to focus the Cyborg Astrobiologist exploration 
system upon the two points that it found most interesting when it was in the more distant tree grove, namely the two 
wet and dark regions of the lower part of the cliff face. By moving from 60 meters distance to 10 meters distance 
and by focusing at the closer distance on the interest points determined at the larger distance, we wished to simulate 
how a truly autonomous robotic system would approach the cliff face. Unfortunately, due to a combination of a lack 
of human foresight in the choice of tripod position and a lack of more advanced software algorithms to mask out the 
surrounding & less interesting region (see discussion in Section 4), for one of the two dark spots (Figure 8a), the 
Cyborg system only found interesting points on the undarkened periphery of the dark & wet stains. Furthermore, for 
the other dark spot, the dark spot was spatially complex, being subdivided into several regions, with some green and 
brown foUage covering part of the mosaic (Figure 8b). Therefore, in both close-up cases the value of the interest 
mapping is debatable. This interest mapping could be improved in the future, as we discuss in Section 4.2. 

The third and final tripod position was at a distance of about 300 meters, near our automobile on the road leading 
to the cliffs. We were careful to choose a mosaic that properly masked out the grass & trees as well as the sky. This 
mosaic ended up being a 1 x 4 mosaic (Figure 6) with a downsampling of 8, which spanned the 10 meter width of the 
hollow in the cliffs, up to a height of about 30 meters. Again, we were pleased when the system found that the dark & 
wet spot near the bottom of the cliff face was the most interesting point. If we had begun the Cyborg Astrobiologist' s 
2nd mission from this 3rd tripod position, instead of immediately going to the shady tree grove, then the sum of these 
results (from the 3rd tripod position to the 1st tripod position; and either discounting or improving the 2nd tripod 
position) demonstrate that the computer vision algorithms inside the Cyborg Astrobiologist could have served as very 
reasonable guidance for the geological approach of an autonomous robot towards the cUff face. 

4 Results 

4.1 Results from the First Geological Field Test 

As first observed during the first mission to Rivas on March 3rd, the characteristics of the southward-facing cliffs 
at Rivas Vaciamadrid consist of mostly tan-colored surfaces, with some white veins or layers, and with significant 
shadow-causing three-dimensional structure. The computer vision algorithms performed adequately for a first visit to 
a geological site, but they need to be improved in the future. As decided at the end of the first mission by the mission 
team, the improvements include: 

• Shadow-detection and shadow-interpretation algorithms, 

• Segmentation of the images based upon microtexture, and 

• Better methods to control the adaptation of the weighting coefficients determined by the Cortical Interest Map 
algorithm (Rae, Fislage & Ritter 1999), which we used to determine our interest map. 

'These dark & wet regions were interesting to us partly because they give information about the development of the outcrop. Even if the 
relatively small spots were only dark, and not wet (i.e., dark dolerite blocks, or a brecciated basalt), their uniqueness in the otherwise white & tan 
outcrop would have drawn our immediate attention. Additionally, even if this had been our first trip to the site, and if the dark spots had been present 
during this first trip, these dark regions would have captured our attention for the same reasons. The fact that these dark spots had appeared after 
our first trip and before the eecond trip was not of paramount importance to grab our interest (but the 'sudden' appearance of the dark spots between 
the two missions did arouse our higher-order curiosity). 
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Figure 5: Astrobiologist & Geologist Ormo is wearing the computer part of the Cyborg Astrobiologist system, as 
Astrobiologist & Roboticist McGuire looks on. On the right, the team of astrobiologists together for a group pose; 
during both of these pictures, the robotic part of the Cyborg Astrobiologist was slaving away, acquiring & processing 
the images of a 9 x 4 vertically oriented mosaic. These pictures were taken during the March 3rd expedition to 
Rivas Vaciamadrid. Note the absence of the black spots near the bottom of the cliff face; these black spots appeared 
sometime after this 1st mission and before the 2nd mission. Photos Copyright: Diaz Martinez, Ormo & McGuire 

In the last case, we decided that due to the very monochromatic & slightly-shadowy nature of the imagery, the 
Cortical Interest Map algorithm non-intuitively decided to concentrate its interest on differences in intensity, and it 
tended to ignore hue and saturation. 

After the first geological field test, we spent several months studying the imagery obtained during this mission, and 
fixing various further problems that were discovered during and after the first mission. Though we had hoped that the 
first mission to Rivas would have been more like a science mission, in reaUty it was more of an engineering mission. 

4.2 Results from the Second Geological Field Test 

In Figures 6-8, we show the original mosaics of the intensity sub-images for the three tripod positions from the 2nd 
mission to Rivas Vaciamadrid. The computer then found the three most interesting points and repointed the camera 
at each of these three points, in order to acquire higher-resolution color images. This is an emulation of a mapping 
& exploration camera being used first to get the "lay of the land", after which a higher technology science instrument 
(i.e. a mapping Thermal Emission Spectrometer or Mossbauer spectrometer) is trained for more time or with more 
resources on the most interesting points in the broad landscape. Figures 9-10 show how the computer arrived at its 
final decision as to where the most interesting points are located, for one of these mosaic images. 

In Figure 6, at a distance of 300 meters, the system found the dark wet spot at the bottom of the 4x1 mosaic to 
be the most interesting, the rough texture at the top of the mosaic to be the 2nd most interesting, and a patchy white 
& tan colored region in the middle of the mosaic to be the 3rd most interesting. The uncommon map for this mosaic 
(not shown) shows that the system found the dark & wet spot to be most interesting largely due the preponderence of 
pixels of uncommon intensity in this spot. There was significant influence of uncommon saturation pixels for this spot 
as well. Furthermore, the uncommonness of the saturation pixels in the dark wet spot (and also the 2nd rough texture 
interest point) is due to the fact that these saturation pixels are outside the common central peak of the saturation 
histogram. For our pixel-segmentation algorithm, when the pixels are outside the common central peak, these pixels 
are often assigned to be in one of the uncommon segments. Figure 6 might suggest that the algorithm found that the 
telephone cable (in front of the rough texture at the top of the mosaic) to be the 2nd most interesting point. However, 
we find that the rough saturation texture itself is responsible for this region being interesting, and not the telephone 
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cable as one might suspect. 

In Figure 7, from the tree grove at a distance of 60 meters, the Cyborg Astrobiologist system found the dark & wet 
spot on the right side to be the most interesting, the dark & wet spot on the left side to be the second most interesting, 
and the small dark shadow in the upper left hand corner to be the 3rd most interesting. For the first two interest points 
(the dark & wet spots), it is apparent from the uncommon map for intensity pixels in Figure 10 that these points are 
interesting due to their relatively remarkable intensity values. By inspection of Figure 9, we see that these pixels which 
reside in the white segment of the intensity segmentation mosaic are unusual because they are a cluster of very dim 
pixels (relative to the brighter segments, with red, blue and green segment-label colors). Within the dark wet spots, 
we observe that these particular points in the white segment of the intensity segmentation in Figure 9 are interesting 
because they reside in the shadowy areas of the dark & wet spots. We interpret the interest in the 3rd interest point 
to be due to the juxtaposition of the small green plant with the shadowing in this region; the interest in this point is 
significantly smaller than for the 2 other interest points. 

A similar analysis of the segmentation, uncommon and interest maps (not shown here) corresponding to Figures 
8a & 8b, shows, for example: 

• The rareness of the two most interesting green- and blue-boxed regions on the periphery of the dark & wet spot in 
Figure 8a is due to their being particularly brighter than most of the other pixels. 

• The peculiarity of the most interesting central white region in Figure 8b is due to its being particularly brighter than 
most of the other pixels. 

For both of these cases, the interesting bright regions seem to have a frothy-white crust, more so than the nearby 
regions which are more tan-colored. 

The wearable computer's analysis of the close-up 2x2 mosaics of the dark & wet spots (shown in Figures 8a 
& 8b) was somewhat disappointing to us, particularly its analysis of the dark & wet spot shown in Figure 8a since 
the mosaic of this dark & wet spot is less complex than mosaic shown in Figure 8b. In retrospect, given the current 
advancement of the image-analysis software, we should have taken the mosaic from a distance of 3-5 meters instead of 
10 meters, in order to focus only on the dark & wet spot, and not upon its periphery. With the current image-analysis 
software, the computer correctly chose to find the periphery interesting because the periphery was the smaller region 
of the image and because the larger and more common dark & wet spot region was rather homogeneous in its hue & 
intensity properties. The dark & wet spot had particularly diverse properties of saturation; this information could be 
useful to develop future interest map algorithms for the Cyborg Astrobiologist. 

More advanced software could be developed to handle better the close-up real-time interest-map analysis of such 
imagery as shown in Figures 8a & 8b. Here are some options to be included in such software development: 

• Reintroduce the Cortical Interest Map adaptive weighting algorithm or similar algorithms (i.e., see Chung et al. 
(2004) for a recent discussion) in order to determine the optimal weighting oithe H, S & I uncommon maps for this 
imagery. This way the system could pay more attention to the Saturation uncommon map if it deems that this map is 
the salient map. 

• Enhance the interest map so that it includes maps of edges or other image features in addition to the uncommon 
maps of hue, saturation & intensity. This could be useful in searching for interesting boundaries or 'contacts' 
between homogeneous regions, which is a standard technique of field geologists. 

• Add hardware & software to the Cyborg Astrobiologist so that it can make intelligent use of its zoom lens. We 

would need to study & develop the camera's LANC communication interface, or possibly determine if the zoom lens 
could be controlled over its Firewire/1EEE1394 communication cable. With accompanying intelligent-zooming 
software, the system could have corrected the human's mistake in tripod placement and decided to zoom further in, to 
focus only on the shadowy part of the dark & wet spot (which was determined to be the most interesting point at a 
distance of 60 meters), rather than the periphery of the entire dark & wet spot. 

• Enhance the Cyborg Astrobiologist system so that it has a memory of the image segmentations performed at a 
greater distance or at a lower magnification of the zoom lens. Then, when moving to a closer tripod position or a 
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higher level of zoom-magnification, register the new imagery or the new segmentation maps with the coarser 
resolution imagery and segmentation maps. Finally, tell the system to mask out or ignore or deemphasize those parts 
of the higher resolution imagery which were part of the low-interest segments of the coarser, more distant 
segmentation maps, so that it concentrates on those features that it determined to be interesting at coarse resolution 
and higher distance. 

• Develop true methods of image interpretation, which can define the dark & wet spot region in Figure 8a as one 
aspect of the image and the periphery of the dark & wet spot as another aspect of the image. Then, determine the 
relationship between these two aspects. Similarly, for Figure 8b, determine, describe and perhaps explain the 
relationship between the several aspects of this image (dark & wet spot, frothy-white region to the lower left of the 
dark & wet spot; green plant to the left of this frothy white area; dead brown plant to the left of this green plant; and 
dry, unvegetated areas in the rest of the image). 




Figure 6: Mosaic image of a four-by-one set of grayscale sub-images acquired by the Cyborg Astrobiologist at the 
end of its second expedition (though we should have acquired it at the beginning of the mission). The three most 
interesting points were subsequently revisited by the camera in order to acquire full-color higher-resolution images 
of these points-of-interest. The colored points and rectangles represent the points that the Cyborg Astrobiologist 
determined (on location) to be most interesting; green is most interesting, blue is second most interesting, and red is 
third most interesting. The images were taken and processed in real-time between 4:59PM and 5:02PM local time on 
1 1 June 2004 about 300 meters from some gypsum-bearing southward-facing cliffs near the "El Campillo" lake of the 
Madrid southeast regional park outside of Rivas Vaciamadrid. The most interesting point (green box) corresponds to 
the wet & darkened area near the bottom of the cliffs; this interest point could have been used to guide the geological 
approach to 30 meters and then to 10 meters to the interesting dark & wet spot by the Cyborg Astrobiologist earlier in 
the day, as represented in the next two figures. 
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Figure 7: Mosaic image of a three -by-four set of grayscale sub-images acquired by the Cyborg Astrobiologist at the be- 
ginning of its second expedition. The three most interesting points were subsequently revisited by the camera in order 
to acquire full-color higher-resolution images of these points-of-interest. The colored points and rectangles represent 
the points that the Cyborg Astrobiologist determined (on location) to be most interesting; green is most interesting, 
blue is second most interesting, and red is third most interesting. The images were taken and processed in real-time be- 
tween 1:25PM and 1:35PM local time on 1 1 June 2004 about 60 meters from some gypsum-bearing southward-facing 
cliffs near the "El Campillo" lake of the Madrid southeast regional park outside of Rivas Vaciamadrid. See Figures 9 
and 10 for some details about the real-time image processing that was done in order to determine the location of the 
interest points in this figure. 

5 Discussion & Conclusions 

Both the geologists on our team concur with the judgement of the Cyborg Astrobiologist software system, that the 
two dark & wet spots on the cliff wall were the most interesting spots during the second mission. However, the two 
geologists also state that this largely depends on the aims of study for the geological field trip; if the aim of the study 
is to search for hydrological features, then these two dark & wet spots are certainly interesting; however if faulting 
between layers is of interest, then the dark & wet spots would not retain as much interest. One question which we 
have thus far left unstudied is "What would the Cyborg Astrobiologist system have found interesting during the second 
mission if the two dark & wet spots had not been present during the second mission?" It is possible that it would again 
have found some dark shadow particularly interesting, but with the improvements made to the system between the first 
and second mission, it is also possible that it could have found a different feature of the cliff wall more interesting. 

5.1 Thought Experiment: Blueberry Fields in the Meridiani Planum 

One of the remarkable discoveries made by the MER team this year is the existence of small, dark & round pebbles 
(a.k.a. 'blueberries' ) in many places near the landing site of the Opportunity rover Meridiani Planum. The Opportunity 
rover was sent to Meridiani Planum because this was the main site on Mars that had been observed by the Thermal 
Emission Spectrometer on the Mars Global Surveyor orbiter to have a large quantity of gray hematite (Christensen 
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et al. 2000). Gray hematite is an iron-bearing mineral which often forms in the presence of water, particularly as 
a precipitate in cold standing water (Glotch et al. 2004). The so-called blueberries that Opportunity discovered at 
Meridiani Planum were also observed to have significant hematite in them, based upon studies from both of the mini- 
TES and Mossbauer spectrometer instruments onboard the Opportunity rover (Squyres, unpublished"^, 2004). There 
is significant excitement in the planetary geology community about this discovery, partly due to the existence of a 
place on Earth (Utah) where similar blueberries have been formed (Chan et al. 2004; Squyres & Athena Science 
Team, 2004; Moore 2004; Catling 2004). 

Such a geological phenomenon suggests a thought experiment that could be applied to our Cyborg Astrobiologist 
system as it currently exists: "If the Cyborg Astrobiologist system was taken to a field covered with Meridiani/Utah 
blueberries, would it find the blueberries to be interesting?" The brief, unqualified answer to this thought experiment 
is: "No". If the blueberries covered the landing site, so that their coverage exceeded some percentage (assume for 
argument's sake that the areal coverage of blueberries at the location of the Cyborg Astrobiologist is 80%), then an 
image mosaic taken by the Cyborg Astrobiologist would be dominated by dark blueberries. Therefore, the Cyborg 
Astrobiologist's uncommon map algorithm, as it now exists, would likely search the remaining 20% of the area for 
the smallest color class of pixels, and it would ignore the blueberries. 

However, if the Cyborg Astrobiologist was allowed to approach the field of blueberries from a distance (i.e., from 
orbit or from a neighboring area without blueberries), then it is probable that the Cyborg Astrobiologist would at some 
point find the blueberries to be new and interesting. This is because blueberries don't cover all of Mars, and so at some 
spatial scale, they only have a small areal coverage. So the qualified answer to the thought experiment is: "Yes". 

Furthermore, we could program more advanced renditions of the software inside of the Cyborg Astrobiologist 
so that it could find the blueberries interesting even if the Cyborg Astrobiologist was not allowed to approach the 
blueberry field from a distance. Such algorithm enhancements could include the texture & edge information sug- 
gested above, as well as a possible algorithm to search for uncommon shapes. With spectrometer information (i.e., 
visible/near-infrared, mid-infrared, Raman, and/or Mossbauer) and with biased searching for particularly-interesting 
spectra, textures, rock shapes and/or rock juxtapositions, the odds of success could also be maximized, either in hunt- 
ing for blueberries or in serendipitously finding such blueberries. 

These advances in our system could take some years to implement, but we will surely implement some of them 
in the near-term. In the meantime, as discussed above, the valued concept of a geological approach will remain 
our solution to the 'blueberry-field' thought experiment. Nevertheless, such a thought experiment does point out a 
significant inadequacy of our simple computer vision algorithm which searches for small regions of the color imagery, 
as discussed in the preceding three paragraphs. Such simple algorithms need to be enhanced and brought into relation 
with other algorithms, and perhaps with a visual memory of past imagery (i.e., the VAMPIRE project (Wrede et al. 
2004)). 

A visual memory would also be useful to record and track changes in the areal coverage of components in the 
scenery, as mosaics or images are acquired from different camera positions or for different outcrops. In a different 
outcrop, the dark wet spots might dominate the whole outcrop, or the whitish color from the gypsum might not 
dominate the outcrop. The visual memory could keep track of outcrops studied in the recent past, and inform the user 
when an unconamon area of a particular image or mosaic is truly novel in areal coverage, when compared with an area 
of similar color or texture characteristics in prior imagery. 

5.2 Outlook 

The NEO programming for this Cyborg Astrobiologist/Geologist project was initiated with the SONY Handycam in 
April 2002. The wearable computer arrived in June 2003, and the head mounted display arrived in November 2003. 
We now have a reliably functioning human and hardware and software Cyborg Geologist system, which is partly 
robotic with its Pan Tilt camera mount. This robotic extension allows the camera to be pointed repeatedly, precisely 
& automatically in different directions. 

'"lecture notes from the course on the MER rovers from Astrobiology Summer School "Planet Mars", Universidad Intemacional Menendez 
Pelayo Santander, Spain 
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Prior to each of the two field-trips to Rivas Vaciamadrid, the Cyborg Geologist system was tested on the grounds of 
our Spanish research institute (CAB). These tests were mostly on a wide prairie field: by winter, this prairie field was 
plowed and full of 3-10 cm diameter stones of different, dull colors; and by summer, this prairie field was full of prairie 
grasses and differently-colored wildflowers. In these nearby field-trips, the Cyborg Geologist has been providing us 
with constructive criticism of the basic capabilities of our image- segmentation and interest-map algorithms. 

As of our initial field-trips to Rivas, the Cyborg Geologist system has accompanied two Human field geologists to 
a site of geological interest (i.e. layered rock outcrops) for more advanced testing of these computer- vision algorithms. 
We plan several more such trips in the coming months. 

Based upon the performance of the Cyborg Astrobiologist system during the 1st mission to Rivas in March 2004 
on the outcropping cliffs near Rivas Vaciamadrid, we have decided that the system was paying too much attention 
to the shadows made by the 3D structure of the cUffs. We hope to improve the Cyborg Astrobiologist system in the 
next months in order to detect and to pay less attention to shadows. We also hope to upgrade our system to include: 
image- segmentation based upon micro-texture; and adaptive methods for summing the uncommon maps in order to 
compute the interest map. 

Based upon the significantly-improved performance of the Cyborg Astrobiologist system during the 2nd mission 
to Rivas in June 2004, we conclude that the system now is debugged sufficiently so as to be able to produce studies 
of the utility of particular computer vision algorithms for geological deployment in the field. We have outUned some 
possibilities for improvement of the system based upon the second field trip, particularly in the improvement in the 
systems-level algorithms needed in order to more intelligently drive the approach of the Cyborg or robotic system 
towards a complex geological outcrop. These possible systems-level improvements include: a better interest-map 
algorithm, with adaptation and more layers; hardware & software for intelligent use of the camera's zoom lens; a 
memory of the image segmentation performed at greater distance or lower magnification of the zoom lens; and high- 
level image-interpretation capabilities. 

Now that we have demonstrated that this software and hardware in the Cyborg Astrobiologist system can function 
for developing and testing computer-vision algorithms for robotic exploration of a geological site, we have some 
decisions to make as to future directions for this project, options for these future directions include: 

• Performing further offline analysis and algorithm-development for the imagery obtained at Rivas Vaciamadrid: 
several of the parameters of the algorithms need testing for their optimaUty, and further enhancements of the 
algorithms could be made. 

• Optimizing the image-processing & robotic-control code for the current Cyborg Astrobiologist system for speed 
and memory utilization. 

• Further testing of the existing Cyborg geological exploration system at other geological sites with different types of 
imagery. 

• Speeding up the algorithm development by changing the project from being partly a hardware project with cameras 
and pan-tUt units and fieldwork to being entirely a software project without robotically-obtained image mosaics and 
without robotic interest-map pointing; with such a change in focus, our algorithms could be significantly enhanced by 
studying many more types of imagery: for example, from human geologist field studies on the Earth, from robotic 
geologist field studies on Mars, and from orbiter or flyby studies of our solar system's moons. 
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A Appendix: Software Implementation Details 

Hue, Saturation and Intensity are computed in NEO by the standard HSI subroutine, which uses the HSI model 
described by Foley et al. (1990). Hue (H) corresponds roughly to the color of a pixel; saturation (S) corresponds 
roughly to the purity of the hue; and intensity corresponds to the total brightness of the pixel. Then each of these 3 
H, S, I image slices is downsampled by a given factor D in each direction {D is typically 2 or 4 or 8) in order to 
conserve memory. 

An important but very technical detail in our implementation is that due to the connmon juxtaposition of pixels 
of widely different Hue (H) in the Hue slice (especially for image regions of low Saturation), we needed to perform 
a type of median filter on the Hue shce in non-overlapping D x D neighborhoods prior to the downsampUng. Such 
median filtering prior to downsampling was not as important for S and I as it was for H. 

The median-filtered and downsampled images are finally combined together into a quasi-mosaic of dimensions 
M X N subimages by simply butting the subimages together. We do not perform any mathematical computations to 
ensure that the edges of the mosaic subimages match well with the edges of the neighboring subimages. 

Following Haralick, Shanmugan & Dinstein (1973), we developed custom 2D 'co-occurrence' histogramming C 
code in NST for pairs of pixels of a given separation and orientation, using min/max stretching for preprocessing prior 
to creating the histograms. For these studies, we chose to histogram pixels separated by 1 pixel width and which 
are horizontal neighbors, but these 2D histograms are separately made (Figure 9) for each of the Hue, Saturation and 
Intensity image slices. For other studies, we have chosen, for example, to make eight simultaneous and separate 2D 
histograms for pixels separated by 5 pixels and by 1 pixel, and with pixel orientations of 0, 45, 90, and 135 degrees. 
Perhaps in the future, we can implement and test something akin to a wavelet analyzer of texture. 

With the three 2D histograms (for H, S & I), we use the standard locate^feaks peak finder in NEO for searching 
for the eight largest peaks in the 2D histogram. After locate^yeaks finds a peak, it masks out histogram bins in a disk 
of a given radius w around that peak, and then it searches the remaining histogram bins for the subsequent peaks of 
smaller amplitude. The values of w were chosen by trial and error based upon previous field work, with the values of 
wforH&S being 15 stretched units, and the value of w for / being 20 stretched units. This standard locate4)eaks 
peak finder may not be very advanced, but it is rather fast and somewhat robust. This robustness is somewhat crucial, 
since the structure of the 2D Haralick co-occurrence histograms often have structures that might confound a less robust 
but more advanced peak finder. These 2D histogram structures in the Haralick co-occurrence method, which might 
confound a more advanced peak finder, include: significant variation in the choppiness (or lack of smooth variation) 
of the co-occurrence histogram as well as significant variation in the width of the true peaks in the co-occurrence 
histogram. Furthermore, the co-occurrence histograms often have only one true peak, which is somewhat larger in 
width than the chosen value of w. In this case of only a single true peak, the system will find the first true peak, and it 
will then find the remaining 7 peaks to be the highest points in the histogram that are more distant than w histogram 
bins from the true peak. This may seem somewhat artificial, but in practice, it is a decent method to search for those 
pixels in the image which are truly unusual and interesting, which is our main task in this project. Nonetheless, this 
peak finding algorithm does merit significant study and possible revision in future work. 

After finding the 8 peaks in a 2D histogram of image pixel-pairs, we reordered the peaks, ranking the histogram 
peaks in the order of the number of original-image pixel-pairs within their w neighborhoods (each bin in the 2D his- 
togram counts the number of original-image pixel-pairs that have the corresponding values of the ordinate & abscissa). 
This reordering may seem artificial, but it was necessary given the nature of the locate^ieaks peak finding algorithm 
that we used. To make the possible artificiaUty clear: the peaks were found based upon the largest remaining single 
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pixel values in the 2D co-occurrence histograms, but the peaks were ranked based upon the numbers of pixels in their 
w neighborhoods. An alternative is to use a more coherent scheme which finds the 8 pre-ranked locations of the w 
neighborhoods with largest numbers of pixels. However, we defer this w neighborhood peak-finding algorithm devel- 
opment for the future since the present system seems to work tolerably well and quickly, and since the artifacts we 
have observed from using the current scheme of peak finding and peak ordering have not yet suggested the urgency of 
such an upgraded algorithm. 

After the 2D histogramming and peak finding, the images are segmented by our NEO progjunit implementation 
(custom C code) of the Haddon & Boyce (1990) method. We throw out pixels that have possible conflicts, unlike 
Haddon & Boyce's more careful work. We handle with some care the pixels on the edges of the subimages in the 
quasi-mosaic. However, in practice, these edge pixels cause the most problems with the image segmentation when the 
motor for the Pan-Tilt unit is taking mosaicking steps that do not match well the optical zoom factor of the camera, so 
that the edges of the subimages are easily visible in the quasi-mosaic. A secondary source of subimage edge problems 
for this image segmentation occurs when there are cumulus clouds in the sky, occasionally occulting the Sun, which 
causes some of the subimages to have different lighting properties than their neighbors. If a pixel and its horizontal 
neighbor have values that sets it in one peak's neighborhood (i.e., peak number s of the 8 ranked peaks of that 2D 
histogram), then the pixel is said to be in segment s of that image. The segmentation maps for that image are therefore 
colored by 8 different colors, with red for the 'most conmion' segment s = 1 (which has the most pixels in it), blue 
for 2nd 'most common' segment s = 2, etc. See Figure 9 for an example. 
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b) 

Figure 8: Two different mosaic images of a two-by-two set of grayscale sub-images acquired by the Cyborg As- 
trobiologist in the middle of its second expedition. The three most interesting points were subsequently revisited 
by the camera in order to acquire full-color higher-resolution images of these points-of-interest. The colored points 
and rectangles represent the points that the Cyborg Astrobiologist determined (on location) to be most interesting; 
green is most interesting, blue is second most interesting, and red is third most interesting. The images were taken 
and processed in real-time between 3:27PM and at 4:05PM local time on 11 June 2004 about 10 meters from some 
gypsum-bearing southward-facing cliffs near the "El Campillo" lake of the Madrid southeast regional park outside of 
Rivas Vaciamadrid. The upper mosaic (a) represents the Cyborg's study of the left-most dark & wet spot of Figure 7, 
and the lower mosaic (b) represents the Cyborg's study of the right-most dark & wet spot of Figure 7. 
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Figure 9: In the middle column, we show the three image-segmentation maps computed in real-time by the Cyborg 
Astrobiologist system, based upon the original Hue, Saturation & Intensity (H,S & /) mosaics in the left column and 
the derived 2D co-occurrence histograms shown in the right column. The wearable computer made this and all other 
computations for the original 3x4 mosaic (108 x 192 pixels, shown in Figure 7) in about 2 minutes after the initial 
acquisition of the mosaic sub-images was completed. The colored regions in each of the three image-segmentation 
maps correspond to pixels & their neighbors in that map that have similar statistical properties in their two-point 
correlation values, as shown by the circles of corresponding colors in the 2D histograms in the column on the right. 
The RED-colored regions in the segmentation maps correspond to the mono-statistical regions with the largest area 
in this mosaic image; the RED regions are the least "uncommon" pixels in the mosaic. The BLUE-colored regions 
correspond to the mono-statistical regions with the 2nd largest area in this mosaic image; the BLUE regions are the 
2nd least "uncommon" pixels in the mosaic. And similarily for the PURPLE, GREEN, CYAN, YELLOW, WHITE, 
and ORANGE. The pixels in the BLACK regions have failed to be segmented by the segmentation algorithm. 



The Cyborg Astrobiologist: First Field Experience 



28 



Uncommon Map: HUF 




Uncommon Map; SAT 




Uncommon Map; INT 




nteresi Map = SuiTi(Uncomm Maps) 





Figure 10: These are the uncommon maps for the mosaic shown in Figure 7, based on the region sizes determined 
by the image-segmentation algorithm shown in Figure 9. Also shown is the interest map, i.e., the unweighted sum 
of the three uncommon maps. We blur the original interest map before determining the "most interesting" points. 
These "most interesting" points are then sent to the camera's Pan/Tilt motor in order to acquire and save-to-disk 3 
higher-resolution RGB color images of the small areas in the image around the interest points (see Figure 7). Green is 
the most interesting point. Blue is 2nd most interesting. And Red is 3rd most interesting. 



